Despite major improvements in solid-state image sensor and digital camera technology, conventional digital cameras may have a maximum photo-signal storage capacity that limits the dynamic range of the particular system. The photo-signal charge is stored on a capacitor within the pixel area. The charge handling capacity is limited by the maximum voltage swing in the integrated circuitry and the storage capacitance within the pixel. The amount of integrated photo-charge is directly related to the time the image sensor collects and integrates signal from the scene, i.e., “integration time.” A long integration time is appropriate for weak signals since more photo-charge is integrated within the pixel and the signal-to-noise of the digital camera is improved. Once a maximum charge capacity is reached, the sensor no longer senses image brightness, resulting in data loss.
Intra-scene dynamic range refers to the range of incident light that can be accommodated by an image sensor in a single frame of pixel data. Two common problems faced by all cameras are scenes with wide dynamic range (WDR), and poor sensitivity in low-light situations. Examples of high dynamic scenes range scenes include an indoor room with a window view of the outdoors, an outdoor scene with mixed shadows and bright sunshine, and evening or night scenes combining artificial lighting and shadows. In a typical charge coupled device (CCD) or CMOS active pixel sensor (APS), the available dynamic range ranges from about 1,000:1 to about 4,000:1. Unfortunately, many outdoor and indoor scenes with highly varying illumination have a dynamic range significantly greater than 4,000:1. Image sensors with intra-scene dynamic range significantly greater than 4,000:1 are required to meet many imaging requirements.
A number of solutions have been proposed to address these issues, including displaying large dynamic range images (e.g., 12-bit images) on lower dynamic-range (e.g., 8-bit) displays. One example of a proposed solution is described in U.S. Pat. No. 7,432,933 of Walls, et al., which applies different tonal and color transformations to each pixel. Other solutions include the addition of sensors that adjust the pixel exposure time, an example of which is described in U.S. Pat. No. 7,616,243 of Kozlowski (assigned to AltaSens, Inc), pixel gain, such as the approach described in U.S. Pat. No. 7,430,011 of Xu et al. (assigned to OmniVision Technologies, inc.), and using multiple-sized photo-active pixels, such as the technology described in U.S. Pat. No. 7,750,950 of Tamara, et al. (assigned to Fujifilm Corporation) to collect WDR images in a single exposure.
Modern CMOS sensors are able to achieve extremely low levels of read-noise, e.g., a few electrons. This provides the ability to sense very low levels of light with excellent SNR (signal-to-noise-ratio). However, as sensors become smaller, with more and more pixels (⅓″ or smaller 1080p sensors, or even five or more megapixel cameras), there comes a point under low-light conditions at which the light signal can still be detected, but the SNR begins to deteriorate (SNR<10, for example, in some portions of the image). Under even darker conditions, one may find that light is undetectable from portions of the image, i.e., image information is entirely lost.
High dynamic range imagery is a serious and frequent problem in surveillance and security video. Consequently, there has been considerable effort expended on trying to solve this problem. In some situations, simple, direct, pixel binning, or more sophisticated, adaptive binning after the signal has been read from the sensor can greatly increase the SNR. Artyomov and Yadid-Pecht (“Adaptive Multiple-Resolution CMOS Active Pixel Sensor”, IEEE Trans. Circuits and Systems, 53(10), pp. 2178-2186, 2006) describe a sensor that can adaptively bin the signal into a quadtree depending on pixel-to-pixel signal level variations in the pixel group. Wardell, et al. (“Multiple Capture Single Image with a CMOS Sensor,” in Proceedings of the International Symposium on Multispectral Imaging and Color Reproduction for Digital Archives, Chiba, Japan, October 1999, pp. 11-17) present a CMOS sensor that can adaptively bin cells and customize individual exposure times. The drawbacks of these approaches are that they can require relatively high total pixel counts.
One example of an off-chip adaptive-binning (or smoothing) approach is the PIXON® method which is described in several U.S. Patents including U.S. Pat. No. 6,353,688, U.S. Pat. No. 6,490,374 and U.S. Pat. No. 6,993,204, among others, which are incorporated herein by reference. A similar approach can be found in Apical Limited's sinter algorithm, which comprises altering area image intensity values of an image according to a dynamic range compression image transform. A description of this algorithm can be found in U.S. Pat. No. 7,302,110 of Chesnokov. The output image intensity is modified relative to the input image intensity according to a local area amplification coefficient.
While helpful, digital noise suppression techniques such as the PIXON® method or Apical Limited's sinter algorithm still cannot sufficiently reduce noise to produce the theoretically best possible performance because they combine the signal from each pixel after the pixel has been read-out. As a result, each pixel suffers its own readout noise, and this read noise adds in quadrature when the signals from the pixels are summed, i.e., SNR grows as the square root of the number of pixels.
A more serious consideration is when the level of light impinging on the sensor is reduced, it will eventually fall well below the sensor readout noise. What is needed is an approach that increases the SNR linearly with the number of pixels that are averaged together. If the signal from the pixels could be combined before readout, the signal from each of the n pixels being averaged would suffer a single read noise, rather than n read noises. While on-chip binning can be performed with CMOS devices, only small numbers of adjacent cells can be combined, especially if a color signal is to be maintained. (See, e.g., Meynants and Bogaerts, “Pixel Binning in CMOS Image Sensors”, EOS Frontiers in Electronic Imaging Conference, Munich, 17-19 Jun. 2009, and Xu, et al., “Charge Domain Interlace Scan Implementation in a CMOS Image”, IEEE Sensors J., 11(11), pp. 2621-2627 (2011.)
The difficulty with combining signals from CMOS sensors is that all of the switching and amplification electronics resides locally in the pixel. This makes the interconnection for the binning quite complicated, especially for color sensors. Accordingly, the need remains for an efficient, effective method for extending the dynamic range of camera systems without unduly increasing sensor (pixel) or interconnection complexity and without introducing additional readout noise.